-
Notifications
You must be signed in to change notification settings - Fork 132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create infrastructure for sample code #331
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just had a couple of small comments, and some thoughts for the future as we build the scripts out.
| ----------- | ----------- | ----------- | | ||
| apply_activist_code.py | Gets activist codes stored in Redshift and applies to users in Van | Redshift, VAN| | ||
| s3_to_redshift.py | Moves files from S3 to Redshift| Redshift, S3| | ||
| s3_to_s3.py | Get files from vendor s3 bucket and moves to own S3 bucket | S3 | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think "vendor" to "own" is maybe what the original script was for, but I think the script can be used for transferring data between any two buckets. I think we can generalize this to just "source" bucket to "target" bucket or something.
"AWS_ACCESS_KEY_ID": "", | ||
"AWS_SECRET_ACCESS_KEY": "", | ||
# S3 (vendor) (note: this assumes a Civis Platform parameter called "AWS_VENDOR") | ||
'AWS_VENDOR_SECRET_ACCESS_KEY': "", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we repurpose as "source" / "destination", we would need to change these environment variable keys.. AWS_SOURCE_SECRET_ACCESS_KEY
, AWS_DESTINATION_SECRET_ACCESS_KEY
, etc.
This is my first stab at creating a new infrastructure for handling sample code. I expect it to need significant edits before merging.
This PR:
Some particular feedback I'd like:
Also if anyone can actually test the refactored sample scripts that'd be swell.